General principles of computing, including computer modelling.
I spent a lot of May 2021 acting as a Section Leader in Stanford University's "CodeInPlace" project, which aimed to teach Python remotely to a very large number of students world-wide (about 12,000), staffed largely by volunteers. It was a great experience and I am posting here some of the general advice I gave to my students.
I saw a number of posts on the CodeInPlace forums from people who are thinking about an alternative career path in computing. I wish you well, because there are, indeed, a great many opportunities: the World is accumulating data faster than we know how to process it, and computation increasingly pervades almost every aspect of our lives. As AI techniques become more prevalent that will only become more apparent. Nevertheless, while software developers are often much in demand and frequently well paid because they are essential to an enterprise, do not expect that any of your non-technical managers will have much understanding of your skills, or visibility of your creations: your beautiful "castles in the air" will be fully appreciated only by you and close colleagues, and you may also find that your contributions to projects are not fully credited. Full appreciation only comes after you leave, and something stops working! (This is a particular problem in academia: promotions go to those who publish the analysis of experimental data and the vital software tool-makers who facilitate their work often do not get joint authorship or even a thank you credit. It is one of the reasons some of the software written by academics may be poorly finished by professional standards: there is no percentage in polishing tools that will then be used by your competitors to burnish their own publication records.)
Here, nevertheless, are some, perhaps slightly cynical, remarks from a person at the end of a rewarding career, which I make so that you can go forward with your eyes open and some realistic expectations.
Like you, I did not start with a formal computing education (very few did when I graduated in 1973). In the early days I was mostly self-taught (we had to be - there was no one to teach us). I was, however, lucky enough to be at Cambridge, which was pioneering the application of computers in science. In fact, in my research field, Radio Astronomy, observations were constrained by the availability of computing power to process data and the field was also driving algorithm development. (Not much changes: it is exactly the same today - but on a very much larger scale.)
Computational physics has, since then, earned me a generous salary, and given me many interesting and challenging opportunities. Even more now than 40 years ago, graduates with numerate backgrounds (e.g. maths, physics and engineering) are still often recruited into computing at a high rate. In the UK, six months after graduating, about 20% of mathematicians and nearly 30% of physicists claim to have employment in IT and computing. Many more, like me, would not claim computing as their primary role: they just regard programming as a tool which supports their work as professional scientists and engineers. In contrast, entry to computing from the humanities tends to be down in the low percentages, but I suspect this is more a matter of inclination, since the capability for analytical thinking is not by any means the exclusive province of the sciences.
It is not just a field for fresh graduates. I once mentored a mid-career teacher who had had enough of the classroom and needed an industrial placement for his MSc dissertation project. I was impressed by his systematic attitude to acquiring new skills (his teaching experience clearly worked both ways) and he subsequently found a job with an academic publisher where his attitude, his teaching background and the recent computing skills he had acquired made him the ideal candidate. The moral here is that previous relevant life experience combined with computing can be very attractive to an employer. Given a reasonable amount of aptitude, employers know they can train to improve skills and knowledge, but attitude, judgement, motivation and general “people” skills take much longer to develop.
I often found it rewarding to work with less experienced people who accepted that they had much yet to learn, and that being an “apprentice” to a “master” was the way to their own mastery. They treated every task as an opportunity to raise their standard of execution. (It is one of the reasons I signed on as a "CodeInPlace" Section Leader.)
In contrast, I disliked working with the “know-it-alls” who did not see the point of acquiring new skills, and often proved to know rather less than they would have you believe (and indeed they were often fooling themselves, as well as recruiting managers). They always wanted to use the same programming language for every job (which was, of course, the first language that they happened to have encountered ten years previously).
I was also wary of the contractor only intent on acquiring new badges for his (yes, in my experience very nearly always his!) CV, who would treat your job as the opportunity to learn (at your expense) the latest snake-oil technology before moving on. Advertising themselves as, for example, competent in Python it would turn out that they had done one course of about the depth of CodeInPlace with little subsequent practical experience.
The "CodeInPlace" Python course (or any other introductory programming course) was therefore just a first step (albeit a very import first step). Knowing the basics of a programming language does not turn you into a software engineer or a data scientist, anymore than learning how to hold a spanner turns you into a mechanical engineer, or learning how to operate a microscope turns you into a scientist.
The next step is developing your “computational thinking” skills: that is, abstract problem solving concepts away from particular programming languages. These skills are highly transferable between languages. I did, in fact, once initiate a Python-based project using a team who had no previous experience in the language - but I had used them before and knew and trusted them as very competent software engineers. Within a month they were using Python like the programming experts they were.
Probably the best way to acquire these higher level skills is working with more experienced programmers and observing the way experts tackle difficult challenges: there is no substitute for learning by doing, and no substitute for having your work critically assessed and helped by an acknowledged master.
That that is not always possible for those who must work in relative isolation. There are, however, a great many books giving excellent advice, and you can also study the productions of experts in the mass of open-source material available via the web. But choose carefully: it is not all built to the highest standards. The best examples make the chosen solution look both obvious and easy to understand. “Hard to understand” may mean it is particularly deep and intricate, using a sophisticated design approach that needs to be taught, but more often it is just a sign of ambition exceeding competence.
The waves of fashion afflict software production more than other areas of engineering: snake-oil and magic bullets are forever being offered: languages come in and out of popularity sometimes for no very good reasons (Python currently rides high, for what I think are good reasons). But there is no substitute for actually taking the trouble to really understand what you are trying to achieve: a language lets you express ideas but does not help you decide what to say. Any electronic or mechanical engineer would say exactly the same.
There is, indeed, a point to learning certain programming languages that you may never use in anger, because they may change the way you are able to think about problems: you are fitted out with new mental tools. Anyone who has completed a course in LISP or PROLOG will always be aware of the advantages of recursive algorithms and “declarative” programming for addressing certain types of problem. (Languages such as Java or Python are "imperative" - we tell the computer what to do. "Declarative" languages describe the properties the solution must exhibit and then let the computer decide what to do.) I have at times, particularly when developing machine learning applications, implemented in Python or Fortran algorithms I first learned in LISP (using the recursive function calls supported by these languages). I even had colleagues who found it worth while building a PROLOG interpreter into their Fortran-based computational physics code for more intelligent and convenient specification of users’ problems.
A programming expert usually has about four or five languages that they are actively employing and will tend to choose the one best suited for a particular application, rather than the same, often inappropriate, tool for every job. (They also have four or five others with which they have some familiarity, and in which they could quickly reach expert level if necessary.)
You may move on from programming in Python, and certainly will if you aim for a professional career in computing, but I assure you that you will forever think in terms of "for x in <an iterable collection>", "while <condition>", lists (stacks and queues), dictionaries (hashes, associative arrays) even if the language does not directly support these features, and you have to implement them yourself.
In the present Covid-19 pandemic many governments are using mathematical modelling, (implemented in computer programs) to inform their critical decisions. Those from non mathematical backgrounds hear these words, but may not be clear about what we can and cannot reasonably do with computer models. At present we do not wish to put our faith in false prophets, but we also need to try to foresee the future course of events as best we can.
The current pandemic has brought computer modelling of one kind into a sharp daily-news focus but it is actually pervasive in modern life. My own background involves nearly 40 years of computational physics in the nuclear industry, where I constructed and used highly complex mathematical models implemented in computer programs to support both the daily operation and the safety assessment of nuclear reactors. I have many peers working in other areas of science and engineering, such as climate modelling or aircraft design, who are all doing essentially similar and vital jobs. I developed both a great respect and a healthy scepticism for the application of computational modelling, and know that you need both sophisticated software and sophisticated processes for using such powerful tools: they can both inform and also seriously mislead you.
A `model’ is something that displays certain selected characteristics of the appearance and behaviour of part of the real world. The `selected’ part of this definition is significant: we choose to capture in the model just those characteristics that are in some sense important for a particular purpose, and ignore features that are irrelevant to that purpose. Mathematical models are equations which constrain the relationships between their algebraic variables such that the values these variables can take behave together like the selected real, measurable quantities. We can, for example, model the behaviour of an aircraft in a flight simulator by taking measurements of the pilot’s control inputs (e.g. stick, rudder and throttle) and then calculate using our model the speed, direction and attitude that a real aircraft would take up - probably, that is as long as we stay within the proven limits of validity of the model. Situations where such models are of high practical importance are frequently complicated because the interesting parts of the world are complicated and we have to solve the mathematics on a computer. Nevertheless, even when we can deploy extremely powerful computers we often have to simplify the mathematics (ignore some of the finer detail in the world) in order to get answers in a reasonable time. (A weather forecast that arrives a day late, for example, is not very useful. One that is 24 hours early may be less accurate, but can still give useful warnings.)
So, even our most sophisticated models (such as those used to predict the weather) still often have considerable simplifications because the real world often works at many different levels of detail and it is usually difficult to handle all of them at the same time. In the case of the weather, for example, we ultimately need to understand everything from the way tiny water drops start to form in clouds right up to global circulation patterns. A great deal of ingenuity and hard work is involved in coping with integrating these scale differences, often very successfully, but sometimes we get thrown into situations where the approximations we use betray us. We have to be ready to recognise and deal with those failures.
There is a sense, therefore, in which none of these computer models can ever produce 100% accurate and reliable predictions but that does not mean they are of no use. They can still be used effectively and safely, as long as you know what you are doing. The professionals spend a lot of time validating the accuracy of their models and learning just how far they may depart from reality and how to recognising when this may be occurring. They learn the signs that show when they can be trusted and when they must be regarded with scepticism. (We often spend far more time doing this than building the computer programs in the first place.)
If we are asked for guidance to support a critical decision, we do not just `press the button’ and pass on the predictions. We worry about whether the outputs are very sensitive to the exact values of inputs. If we are feeding real-World data into the models (which is often intrinsically uncertain) then we had better give a range of predictions, based on the the possible variations in the data, and try to work out which of those we can consider improbable and which are more likely. The models, however, do not make the decisions for you: that requires judgement about the amount of risk people are prepared to accept from the potential adverse consequences that cannot be ruled out, and the costs of mitigating those risk and whether those are acceptable. These are human and perhaps political issues.
Even highly oversimplified mathematical representations can be useful because they give valuable qualitative insights into the way complicated systems might behave in a broad sense. In particular we can often quickly gain a good understanding of the way the answers would change when we modify the inputs to the model (i.e. adjust this parameter down and that output goes up). In epidemic modelling, for example, even the simplest schemes suggest that infection rates build up, and then die away as more of the population become immune and it become less likely that an infected person can contact new hosts who have no immunity. They will tell you clearly that reducing the probability of person-to-person transmission may not always reduce the total number who eventually become infected (though it might) but it will certainly spread them out in time and that may be helpful in protecting an overstretched health service. If, however, we want to know exactly when the peak will occur and how high it will be, that requires a more sophisticated approach, looking in more detail at how different groups of people in the population interact with each other.
Physics modelling is easy compared to trying to predict the way people will behave. In order to model epidemics we need to represent the way people interact and we cannot know how every individual is going to behave at any particular time, so one has to divide the population into broad groups (perhaps children, teenagers, young singles, older married with children, elderly, people who live in high-density environment, and so on) and then make broad assumptions, based on empirical observations, about the way each group on average will behave and interact within itself with the other groups.
In principle, you might think, that by adding more detail (such as more finely distinguished population groups) we might get better predictions, but that is not necessarily the case because the finer the detail the less we may know about the average behaviour of a particular group. One may, for example, have reasonable data on the average number of personal interaction of single 20-somethings each day, but further dividing that group by, say, educational attainment or social background we would soon find that only a few individuals have been observed to produce each average: it becomes statistically unreliable.
Furthermore, even as an epidemic progresses, the original data about population behaviour is less reliable because the epidemic itself changes behaviour. We have little empirical data on responses to pandemics in our time and our culture. Reasonable assumptions, will need to be made and tuned to reality as we learn more during the progress of the epidemic.
In fact, as we add more complexity to our models we also raise the risk that we make more mistakes in the mathematics and introduce more errors into the ever more complex software implementation. More complicated computer models also then become much harder to test and validate because they have much more complicated behaviour, and it will cost a lot more in money and time to collect the evidence confirming that all the nuances of behaviour are correctly represented. In fact, the business of testing and validating a computer model to a state where it is sensible to base important decisions on its predictions may often take far more effort than the development and programming of the model. So while, complicated models may potentially produce higher fidelity representations of the real world, they may also go so badly wrong that they are highly misleading.
When we need to be really sure that we are not being misled, it is sometimes better to stay with the simpler but transparently understandable approaches, and build in extra clearly defensible safety margins. If I cannot constrain the magnitude of potential errors in a more sophisticated approach then, although the predictions may be potentially more accurate, I may find that the larger safety margins I must apply actually offer less scope for manoeuvre.
When avoiding errors is really important we often use independent computer models written by different teams to study the same problem. If fact, systematic comparisons of this type suggest that many computer models (even those produced after high investments) may rather be less dependable than their authors might like to believe - see Hatton (1997).
Even then, it is usually not the scientist of engineer who has to make the high impact decisions and communication of advice to senior managers or politicians in a way that does not mislead requires care. The decision makers rightly claim that the balancing of risk and benefits though their choices is their particular province and responsibility. Our responsibility as modellers is to correctly communicate not only our projections but the uncertainties in those projections so that the decision makers can make a fair assessment of the real risks they may need to accept.
The modelling exercises carried out to inform the pronouncements of the International Panel on Climate Change are an excellent example of the way things should be done. Not only do we see contributions from many independently developed models. We also see that many variations of the model are studied, in which the controlling parameters (some of which represent uncertain aspects of the real world) are varied to examine the effect this has on outputs.
In the case of the current climate emergency, you may, nevertheless, be forgiven for thinking that political leaders have so far guided their decision making based on the more optimistic outcomes amongst the predicted climate scenarios - but they have certainly not been misled. I have noted that in the current Covid-19 pandemic (as with the climate emergency) that journalists have often focussed on either particularly optimistic or particularly pessimistic predictions: neither on their own is especially informative.
We should not, however, be hypocritical, though as most safety professional realise, the public understanding of risk is not particularly sophisticated. We all like to think that we put a high value on life, but many of us still climb into cars and accept the not insignificant risk that we will kill or injure ourselves, our family members or even strangers who happen to get in the way. Safety professionals and their political masters, working on our behalf, have the unenviable job of working out how much inconvenience and cost we are really prepared to accept for improved safety, while still allowing us to believe one thing and do another.
None of this is easy or quick, and it often involves a good deal of informed insight, which is why it is important that such complicated models are operated by `suitably qualified and experienced’ engineers and scientists. In the end it is the scientist or engineer who makes a judgement as to which predictions should be trusted. If the computational models have been developed, built, tested and validated by suitably qualified and experienced people, using the appropriate techniques compatible with accepted professional standards, and they are operated by suitably qualified and experienced people within the scope of the validation database, it is usually possible to justify a claim that any remaining errors still in the software probably have an insignificant effect on the outcomes. (Conversely, we would probably also claim that a modelling defect that is having a serious undesirable influence on outcomes would most likely be noticed by our highly qualified users.) If any of these conditions are compromised then there must be a corresponding decrease in the amount of trust that we should give to the results. The ability to successfully communicate these subtleties to decisions makers is also part of what it means to be suitably qualified and experienced.
Given the importance of computational modelling in the modern world you might think that there is a consensus about the skills and qualifications required, and the professional standards that ought to be enforced. This is a surprisingly grey area, though regulated industries, such as nuclear, have to establish and show they comply with processes and standards that will pass audit by the appropriate authorities. (I once had some responsibilities to my former employer both for defining appropriate working practices and the training requirement that ensured that our `suitably qualified and experienced’ staff were able to operate them successfully. I spent a good deal of my time looking at what engineers in similar industries doing similar work considered reasonable…and constantly worrying whether I had found the correct balance between regulation and trust in the competence of the often highly innovative people who looked to me for advice.)
In a wider context there are a number of reasons why it is very difficult to produce agreement, ranging genuine well-founded disagreements about choosing appropriate techniques in very disparate contexts with very different risk spectra. There are also real difficulties in finding enough of the rare people who can exhibit the full range of required skills and knowledge, including, for example, high levels of ability in physics and maths combined with extensive knowledge and experience on computer science and engineering and the essential interpersonal skills to understand and communicate the human context of everything we did. We also have to deal with sufferers from the Dunning-Kruger effect: (see Wikipedia!) incompetent people often fail to recognise their own incompetence, and it is not unknown for their undoubted abilities in self-promotion to push them to positions of responsibility where they can be a block to raising standards. Those of us who have had extensive training and experience in both physics and computing all too frequently recognise the signs.
It is about finding out who you can trust as well as which models you can trust.
The author is a Fellow of the Institute of Physics, a Fellow of the Royal Astronomical Society, a Member of the British Computer Society and also a Chartered Engineer.
References
Hatton, L., 1997. The T-experiments: errors in scientific software. In Quality of Numerical Software (pp. 12-31). Springer, Boston, MA.