Maybe it’s not you, it’s the water

The best soccer teams never let the ball get to the goalie, and the healthiest goldfish still need their water changed. These were the central themes of Dr Sandro Galea‘s speech Thursday at the Canadian Society for Epidemiology and Biostatistics’s national conference in Winnipeg.

The talk highlighted the over-focus on tiny, proximal parts of an overall chain of events leading to disease. By now we have learned everything from eating nuts to vitamins to swimming prevents cancer, for instance. We have also made mass investment into personalized medicine, taking tailored health to a new level.  The message is clearer than ever: health is experienced by the individual so it must be in the individual’s hands.

At the same time, most of us are grounded in an overall interest to improve population health. By focusing on the determinants of health at an ever-increasing individual level, we ignore the systems and environment within which people make health decisions. For instance, there has been a major decline in automobile accidents over the past century. What is responsible? Safer roads and safer cars.  Not focusing on the individual driver’s abilities. Similarly, having a population impact requires a perspective on the conditions under which we can learn to eat more nuts and vitamins, and buy more swimming pool memberships.

IMG_20160609_101301

Our own Sofia asking Dr Galea a question

According to  Dr. Galea, the solution centers on a re-calibration of time and monetary investment. This does not involve spending more time and money, but pulling existing research energy and health system funding towards evidence on population options and public health infrastructure. The ‘no net increase in spending’ argument should win over politicians who may be completely unaware of this perspective; lately, funding has gone the opposite way (while interesting in theory,  imagine the public reaction to the headline, “Funding cuts to hospitals,” regardless of the overall benefit).

Ultimately a population approach stops people from getting sick in the first place: it is all the offense and defense in place before the ball ever gets to the goalie.  It also acknowledges the limits of individual responsibility: a goldfish eating the healthiest and exercising the most can only go so far in murky water.   While we still need a goalie when we get sick, and we still need to look after ourselves, stressing only individual health determinants hinders fulfilling what most of us would like to see: better population health.

 

Thinking about observational studies like RCTs

You too can conduct a high quality, valid observational study!  Today, Dr. Miguel Hernan reminded us of some overlooked aspects of doing exactly this.  When done correctly, we might get close to an estimate from a  randomized controlled trial. When done incorrectly, we probably have not shaped our thinking in the right way. Ultimately, sometimes observational studies can mimic RCTs (without replacement). To even begin to try to achieve this, we need to approach our observational studies more like RCTs.

The thinking begins by asking an answerable  question.  Questions for RCTs involve interventions that are ‘randomly assignable’.  ‘What is the effect of taking aspirin compared to a placebo on Y?’ ‘What is the effect of giving people a basic living wage compared to targeted benefits on Y’? Inherent to these are a well-defined intervention that is the same across participants.  In contrast consider, ‘What is the effect of weight loss on heart disease?’  This is not a causal question because multiple interventions lead to weight loss (did you start exercising or start smoking? The effect will vary depending).

Next, consider eligibility and Time 0. Here is where even high-quality observational studies can go bad. Common mistakes are accidentally (?) including prevalent users of an intervention rather than incident users or inducing immortal time bias.  A great example was the RCT versus observational design controversy regarding hormone replacement therapy and partial resolution once the time 0 and prevalent users inclusion issues were addressed.  To avoid such blunders, get to know your data. Very well.  In contrast to an RCT where we might build the data from scratch, us observational study-ists purchase pre-existing data and are thus at the mercy of its documentation.  Thinking like a trialist and digging into the data reveals whether it is possible to answer your causal question, or some worthwhile version, after all.

Third, minimize confounding biases and consider estimating a per-protocol effect while you’re at it.  Eliminating these biases means that the only important difference between treatment and control groups is the treatment itself. There are methods such as inverse probability of treatment weighting, propensity score matching, etc, to help mimic random assignment. Implementing these methods using baseline covariate measures allows estimation akin to an RCT’s ‘intent-to-treat’ (ITT) effect (the effect of being assigned versus not assigned to treatment).

But remember, confounding is an issue in both RCTs and observational studies if studying the ‘per-protocol’ effect (the effect of actually taking versus not taking a treatment).  In RCTs this problem is generally dealt with by ignoring it. Estimating a per-protocol effect requires more complex analyses using time-varying exposure and covariate measures.  But given per-protocol estimates are usually the actual effect of interest, and ITT estimates can be problematic, more effort should be made to present both ITT and per protocol results.  Dr Hernan predicts these efforts will become commonplace in the near future.

5 Software Tools to Make Grad School in Epi Better

As epidemiology and public health graduate students, a good number of us spend almost more time on computers crunching data than watching youtube. We all have our favorite data analysis tools installed: R, Stata, SPSS, SAS, JMP, WinBUGS, Matlab… we use Dropbox to sync and backup files, Google Docs to collaborate, Endnote or Papers to manage our PDFs and citations, and Evernote to manage our notes.

gif_huggingmonitor

Friday afternoons in Purvis Hall

But aside from the famous  tools we all know and love, there are a lot of awesome software tools and plugins out there that can make our lives just a little easier. You want to search for 50 different keywords in 50 different windows at the same time? there’s a plugin for that (Chrome). You want to download citations on-the-go? this button is mandatory (Chrome, Firefox).  You want to force that window to stay on top so you don’t have to flip back and forth? download a little utility software (Win, Mac).

Here are 5 software tools that have made my life just a little bit better:

Continue reading

Exciting news: Unique data on social policies is now available

The social and economic conditions that surround us can affect our health.  This is not a new idea.  If the notion was not broadly appreciated by the time it was formalized in the Lalonde report, the point was hammered down more thoroughly in the Marmot review.  However, with little evidence quantifying the health impact of policies meant to address these conditions, this idea has large stayed as just that – an idea. Due to their complexity, we have been rarely able to answer questions like, ‘What exactly would happen to people’s health if we passed policy x, y, or z?’ or ‘How many less  people would get sick?’

Part of the challenge is the lack of analyzable policy information. The other part is the inherent difficulty in using conventional epidemiologic methods to answer such questions.  Taking on these challenges, the McGill-based MachEquity project (2010-) has been building databases and applying robust methods in an effort to build a slew of causal evidence of social policy effects on health in low and middle income countries. Below is a summary of their work and recent launch of data for public use.

How has MachEquity built ‘robust’ evidence?

First, there is the stringently-coded policy data. Focusing on national social policies, staff looked at full legislation documents in each country, amendments and appeals, or secondary sources if original documentation was not available.  Two researchers then quantified the legislation into something meaningful (for example, the months  paid maternity leave legislated), ultimately resulting in longitudinal datasets for with policy information from 1995-2013.

Second, the group is making use of quasi-experimental methods that attempt to mimic random assignment.  The latter is the ‘gold standard’ to evaluate the impact of anything (policy, intervention, drug…) because it ensures that we eliminate all other potential explanations for any differences we see in the people’s outcomes (e.g. their health status post-experiment).  Evidently, perfectly controlled randomized experiments are most often impossible when we are dealing with social determinants of health (can we randomize people to have more education?).  Enter: quasi-experimental methods. There are entire books and courses on this (literally – look here), but the basic idea is that we can mimic randomization by eliminating other sources of variation, e.g. over time, in different types of people, or in different countries, by controlling for it in specific ways.

So what have they done so far?

Built policy databases. Published. Presented. A lot. See here.  But not only that, researchers and staff on this project work in close collaboration with partners at the Department of Global Affairs, non-governmental organizations such as CARE, and ‘package‘ their work for policy-maker audiences . After all, the actual policy-making is in their hands, which are often far out of reach from academic research. Specific research topics have included the effect of removing tuition-fees, health service user-fees, maternity leave legislation, minimum age-of-marriage laws on outcomes like vaccination uptake, child mortality, adolescent birth rates and nutrition.

Where is the policy data and how can I use it?

Policy datasets on maternity leave, breastfeeding breaks at work, child marriage and minimum wage are now available for download here!  For each determinant, longitudinal data on low and middle income countries’ policies is available. You can therefore make use of quantified social policy information and changes in legislation over time, and the infinite possible analyses that such data lends itself to.

What do we do with all this data?

According to some, ‘big data’ will transform everything, infiltrating every aspect of our work, play and comings and goings.  But what are the implications for epidemiologists? What exactly is ‘big’? What exactly is ‘transform’?  What’s next for us?

Daniel Westreich and Maya Petersen addressed these questions in the Society for Epidemiologic Research’s digital conference  today.  For epidemiologists, the consensus (from those keen to type responses in the chat box) was that big data may not be as revolutionary as popular imagination suggests.  However to take full advantage, we may require new methods, training, more collaboration with programmers and ultimately better PR.  Below is a full summary of the talks.

So what is ‘big’?  It depends.

Daniel Westreich quoted others in saying ‘big’ is a moving target: what is big today was not big many years ago (think of your first CD compared to your current iPod).  The summary I liked best: ‘big’ is anything that cannot fit on conventional devices.  For example, I only discovered my dataset was ‘big’ when I tried to read it into R, the program froze, and my computer crashed.  That’s big data (or a bad computer, but anyway, that’s the idea).

And could ‘big data’ transform epidemiology? Sort of.

First, unfortunately, simply having more data does not guarantee that causal assumptions are met. For example, Dr Westreich explained how scrapping big data from Twitter would result in huge amounts of highly biased data because the site is only used by a non-random 16% Americans.  At the opposite extreme, we may end up over-confident in highly precise yet biased results. Big data could instead contribute more to prediction models. But Maya Petersen cautioned that even in these models, our implicit interest is often still causal – how often are we interested in knowing the probability of an event without even taking guesses as to why it occurs?

At the same time, we would need to move beyond classic model selection procedures to use it.  Imagine 1000s of possible covariates, interactions and functional forms.  According to Dr. Petersen, the way to arrive at a logical estimator must be to move away from using our logic: take humans out of it.    She gave examples using UC Berkley’s signature SuperLearner in combination with Targeted Maximum Likelihood Estimation. Essentially, the first amounts to entering the covariates in a type of black box that attempts to find the best combination. Obviously the ‘best combination’ depends on the question at hand, hence the combined use with Targeted Maximum Likelihood Estimation.  Though a specific example, we can only expect  the use of such computer-intensive methods to increase alongside the use of big data in epidemiology.

Finally, what’s next for us? Training, collaboration, PR.

1) Revised Training: Requiring the use of these more computer-intensive methods also requires development of more advanced programming skills.  But both speakers commented on the existing intensity of epidemiology PhD training.  In fact, we are perhaps the only discipline where students come into the PhD with exactly 0 previous epidemiology courses. There is a lot to learn.  At the same time, we cannot place the onus entirely on the students to self-teach. A better solution may be more optional courses.

2) Better collaboration: Rather than all of us returning to complete Bachelors in Computer Science, we could just become friends with programmers. In fact, there are lots of them.  Dr Petersen discussed how teaching collaboration with computer scientists is a more feasible approach than teaching computer science itself.  Part of that involves knowing the kinds of questions that we need to ask programmers.

3) More PR: Epidemiology public relations is a little non-existent relative to others sometimes (e.g. Economists).  If we think we can benefit from big data to answer population health-relevant questions, we need to get ourselves invited to bigger discussions on the topic. For example, epidemiologists should be involved in a discussion on what data need to be collected. But the status quo generally excludes us.

More information: Daniel Westreich / Maya Petersen / Big Data in Epi Commentary

Operational research: bridging theory and practice

One of the things we are told as students in public health and epidemiology is that our work has real life implications and will help in making better decisions in practice. On the first week of May a group of us from very diverse backgrounds, academia and field workers, participated in a week-long course focused on operational research methods offered through McGill’s Global Health Programs and partners. This course gave us a chance to see how exactly that gap between academia and practice can be bridged. Operations research is a term with broad scope, used by the military, industry, and the public sector. The objective of this course was to give us insight as to how analytic methods can be used to guide planning and decision-making in global health operations, particularly in low and middle income countries. The workshops were guided by Dr. Rony Zachariah and Dr. Tony Reid of MSF, Dr. Ajay Kumar of the The Union, and Dr. Srinath Satyanarayana of McGill University. Below are some ideas worth sharing that participants in the course from our department picked up:

Ebola treatment unit (ETU) run by Médecins Sans Frontières (MSF). Photo: UNMEER/Simon Ruf released under creative commons.

Ebola treatment unit (ETU) run by Médecins Sans Frontières (MSF). Photo: UNMEER/Simon Ruf released under creative commons.

The simplicity of operational research: simple solutions for important issues – Vincent Lavallée (Public Health)

Like many others, I was new to the field of operational research when beginning the course. My greatest takeaway from this course was the potential for simple solutions when tackling difficult questions. A common trend coming from academics is a stubbornness that demands perfect study design. This is often described as the holy grail of epidemiological research, the randomized control trial. While it is very important to identify potential biases and errors in reporting when conducting a study, unfortunately gold-standard RCTs are rarely feasible in the field.

What I enjoyed most about this course was how they highlighted the use of natural experiments and creative solutions to answer questions regarding health care implementation and utilization in low resource settings. While finishing my public health degree, one class required we write proposals for theoretical research projects. Many groups often got caught up in trying to answer all the questions, resulting in increasingly complex study designs. It was refreshing to see how operational research teams from MSF take on one or two very poignant questions and develop simple yet eloquent solutions to answer them. In doing so, they manage to change policy and current practice in these settings.

The importance and challenges of publication in operational research – Marzieh Ghiasi (Epidemiology)

One of the interesting topics covered in the course was the important role that publication can play in operational research. In academia, for better or worse, the mantra ‘publish or perish’ exists in part because publications are a measure of productivity. In implementation settings, the objectives and pressures are different and publication is not a priority. In fact, projects are often are implemented by governments and agencies, without a strong empirical framework or post-hoc analysis– and the people doing the implementation may or may not be trained in constructing scientific publications. The course instructors highlighted how conducting operational research and publishing can play the role of providing an evidence-based road-map and dissemination tool. Consequently, the capacity to conduct operational research is built by not only by training people how to develop protocols, collect data, but also how to publish and do it well. The presenters gave the example of a course by The Union/MSF focused on developing these skills.

We had a hands-on overview of how to use EpiData, a free open software for systematic data entry ideal for use in constrained settings. As well, an overview of how the publication process works: for example, the often overlooked but important task of actually looking at and adhering to author guidelines before submitting a manuscript to a journal! One of the most interesting things I took away from the workshops was the idea of ‘inclusive authorship’ in operational research, which is critical in projects that involve dozens and dozens of people in design, implementation, data collection and analysis. The instructors recalled their own experiences of trying to chase authors and contributors down by email versus bringing dozens of people in a room over the course of a couple of days to get them to write a paper together (the latter works better!). Bringing 30-something people to write a paper is, of course, in itself an operational challenge. But, as this paper showcases, it is possible and should be done to ensure fairness and engagement.

The untapped potential of operational research – Marc Messier-Peet (Public Health)

When I first glanced at the course outline for this Operational Research course, I felt this wave of relief come over me. Yes, people are researching implementation science, and yes, people acknowledge the potential gains it can bring to the field of public health. Delivered by an exceptional team of operational research experts, we had an excellent crash review that would appeal to anyone interested in strengthening health systems. Among the many things I took away were how to improve routine data collection through streamlining it and making it as user-friendly as possible, in order to ensure benefits for researchers and decision makers alike. We were shown that data collection is not inherently justified in itself. In operational settings, there is an ethical imperative as publicly-funded researchers to make sure any data collected answers a relevant question and the final work is disseminated to those best suited to use it.

Focusing on collaborations and partnership between stakeholders, the course underlined how it is important to build relationships all along the operational research trajectory. With a greater emphasis being placed by the international development community on impact evaluation and donor accountability, operational research can help find the necessary tweaks and adjustments needed to improve any under-performing health systems. Perhaps we in Canada could benefit from turning the operational research lens inwards, and develop our capacity to see how our institutions could perform better? The questions raised from an operational research approach are ones that need to be asked, and provide the opportunity for engaged researchers to bridge the ‘know-do gap’ and see their work make a real difference in people’s lives.

Why is society the way it is? The problem of infinity DAGs

Consider two questions:
1. Does racial bias in law enforcement in the U.S. occur? (assuming the answer is yes)…
2. Why does racial bias in law enforcement in the U.S. occur?

Ezra Klein wrote a piece in December on the danger of controlling for large numbers of variables in analysis because we could end up ‘controlling out’ key parts of our effect of interest (no, he doesn’t appear to be an epidemiologist or even any type of researcher, but nevertheless seems to have a better understanding of confounding than many with those titles).

In DAG language:
healthyinferenceIsvWhy_jpeg

As he aptly recognizes, researchers know this. As we’ve been taught, there are also ways of dealing with it: base your variables on substantive knowledge, do not adjust for mediators, and if you do adjust for mediators, know what you’re doing [see: mediation analysis].

Klein’s problem with over-controlling is philosophically grounded in question number 2 above. He suggests that controlling for effects of the exposure prevents us from knowing why phenomenon occur. Once you control for location of drug use, black people end up far less likely to be arrested for drug crimes than white people. This is because they are more likely to use drugs in urban settings, and police are more likely to make arrests there. So in controlling for location, we lose the ‘why.’

But there is a distinction between question 1 & 2.  The first is complicated enough: teasing out whether an association exists and its strength is undoubtedly ‘epidemiology.’ It’s also quantitative ‘sociology’ with some ‘economics,’ and probably any science. It involves describing the world as it is.

Consider the (highly simplified) reality of why people who are black are potentially more likely to be arrested (Y= e.g. Arrest):

healthyinfWhy2

There are lot of cumulative, intertwined reasons ‘why’ racial bias might exist in U.S. law enforcement.  The particular letter (i.e. variable) we choose to study is somewhat arbitrary (A on Y? B on Y? C on Y?…).  Say we look at the effect of C on Y.  There are ancestors (A and B…) and mediating effects (D and E….).  Such is the case no matter where our study sits on the causal path.  In other words, there are infinity letters behind and ahead of our letter of choice.

Figuring out why society is the way it is is entirely relative. When viewed as ‘yes’ or ‘no,’ (‘Have you ever been target of racism?’) we can measure these things. But the cut-point is arbitrary.  When viewed as a cumulative sum of experiences, the DAG possibilities approach infinity; the ‘why’ is less and less measurable (‘So, what’s like to be Black in America?’).

As Klein suggests, we shouldn’t over-control or adjust for mediators. But perhaps the problem with this is more to do with biasing our analysis away from some true effect (i.e. the effect of the letter we arbitrarily choose to study) than Ezra Klein’s suggestion that it prevents us from knowing why. Are epidemiological studies of social phenomenon meant to answer ‘why’? Can they? The rigour in their methods comes from their ability to figure out what is. We know black people are more frequently arrested for similar crimes to white people. Why? We can only adjust so much, calculate the effect along so many possible pathways, and collect so much data. And we probably still wouldn’t know.

Open-access publishing – the good, the bad, and the ugly

This post summarizes a seminar given by Dr. Madhukar Pai at the Lady Davis Institute on February 2nd, 2015 entitled “Open-access publishing – the good, the bad, and the ugly”. A copy of the lecture slides were kindly provided by Dr. Pai and are available here (PDF).

As students and researchers, most of us have tried to access a scientific article important to our work only to be confronted by the publisher’s pay-wall. This is even more frustrating if the work we are trying to access or use is our own. Unfortunately, the body of scientific literature upon which we rely is primarily published using this “pay-wall” or “pay-for-access” model. This week, Dr. Madhukar Pai discussed open-access publishing as a worthy alternative while giving practical insight into its shortcomings and the outright predatory practices that have followed its introduction. For those unable to attend the talk, this post summarizes some of the important points discussed.

In the conventional publishing paradigm, researchers sign away the rights to their articles to publishers who exclusively provide access to readers for a fee. Publishers achieve high profit margins with this model while universities struggle to afford the exorbitant cost of subscriptions. For those without institutional access, results from high-quality scientific research are prohibitively expensive, disproportionately affecting researchers in low or middle-income countries. In clinical medicine and epidemiology, the public-at-large are the population for whom the research is conducted and yet they have the least access to the scientific findings. This is particularly questionable given the amount of this research that is publicly funded. Recently, high-profile scientists and institutions have joined the voices of criticism against traditional scientific publishing calling for a more open alternative. Many public granting agencies (e.g., CIHR, NIH) and private donors (e.g., Bill and Melinda Gates Foundation) are also now demanding that results of funded research be publicly accessible.

“Open-access” (OA) journals have been touted as the new alternative to conventional publishing. It stands for “unrestricted access and unrestricted reuse”. Under paid access, authors could be restricted from making copies of their own research available to colleagues or students, and could be required to get permission and pay additional fees to use parts of their work in other publications. In principle, OA eliminates such restrictions creating a freer dissemination environment with accelerated discovery, improved education, and public inclusion. Practically, there are different “shades” on the OA publishing spectrum. For example, some journals provide immediate and unrestricted access, while others use a hybrid approach where access requires a fee for a limited period of time, after which it becomes free.

Despite the potential benefits, open-access has its own shortcomings which have become apparent as OA journals have become more widespread. OA publications suffer from uncertain stability, which has been highlighted by shut-down publications. Additionally, these journals profit from accepting articles rather than providing access, meaning that charges for submitting authors are often substantial.

If we believe that the role of journals is to act as gate-keepers, then a fundamental conflict of interest is being created with respect to article quality when journals are paid based on how many articles they can publish. In the world of “publish or perish”, there has been no shortage of eager and outright predatory entrepreneurs vying to profit by publishing anything and everything that gets submitted.

While there are a number of major respected OA publishers (e.g., PLOS, BioMed Central), the number of dubious and so-called “predatory” OA publishers are on the rise. Predatory publishers operate by accepting fees to publish whatever is submitted, with little or no review, in order to make a profit. The worst predators are bogus operations that rely on spam-like tactics (including unsolicited emails and invitations), usually headquartered overseas. With predatory publishers corrupting the landscape of open-access, it is becoming increasingly difficult to identify reputable ones. Scholarly Open Access is an online resource created and maintained by Jeffrey Beall which systematically identifies and lists dubious/predatory publishers and journals (“Beall’s List“).

Other than checking Beall’s list, here are a few red flags for spotting a predatory journal:

  • unsolicited emails with spelling and grammatical errors,
  • journal/conference is completely unrelated to your field,
  • e-mail domain is generic (e.g., gmail or hotmail),
  • promises of rapid publication, reduced fees, or discounts,
  • lofty titles that mimic established journals (“International Journal of …”),
  • editor or editorial board names not provided,
  • fictitious or missing publisher address.

Take-home messages from the talk

Open-access publishing is here and there are great reasons to support it as an alternative to traditional pay-wall access. Regardless of the access model, a strong peer-review system will remain critical for ensuring the quality of scientific publications. The sustainability and credibility of publishers are potential challenges for open-access, particularly given the rise in predatory publishers. It is important that we are vigilant about identifying and avoiding predatory publishers both in order to protect ourselves and to stop incentivizing their practices.

Sources and additional reading

What should I do with my life? PhDs & PostDocs         

Drs. Nicholas King, Ashley Naimi, and Serene Joseph all gave their thoughts on deciding next life steps last week. They described their experiences as PhD students and post-docs in a session for students in the department on Thursday, January 29.  They had a slew of advice to guide choices that lots of us are contemplating.

General advice to make decisions:
-Picture yourself in 5 years walking into your dream job. What are you doing?
*solving problems
*thinking, writing
*working with people
*working in an office
-Your vision of where you want to be is the best guide to deciding your next move.

-There is no direct path to achieving your dream job. Are you:
*committed to one subject
*interested in several things
*ready to switch your area of expertise
-All are paths that take varying lengths of time to get where you want to be, but none are wrong.

-No matter what you do/don’t do, there is a sacrifice.
*Doing a PhD: Sacrifice income, savings, benefits, and settling other parts of your life
*Not doing a PhD: Sacrifice certain leading positions or promotions because you might hit a glass ceiling

On PhDs:
-Academia is not the only end goal
-Don’t feel you need to ‘know everything’ on entry: it’s not expected that PhD students know how to ask scientific questions or design methodology. Learning this is the point of a PhD.
-If you come in with a solid idea, be open minded: you might miss out on some pretty cool opportunities to learn from professors and work on existing projects
-Don’t go for the ‘hot’ topics for that reason alone: It’s tempting to study current flavours of the month, but genuine lack of interest will become apparent when you compete against people who are actually passionate about a topic for future positions
-Do other things: It’s important to get your PhD done, but enriching to do side RA projects/TAships. You get exposed to others’ work and styles.
-Publish, but not too much: This is a catch-22. If you’re too focused on publishing, you’ll miss out on the unique PhD student opportunity to read, learn and develop your own ideas.  If you don’t publish enough, it is tough to get future positions.
-Through your PhD, keep a running list of research ideas that you don’t have time to look at. A postdoc might be the perfect chance to explore these.

On Post-Docs:
-Really only necessary if academia is your goal
-Start applying at least 1 year before you plan to defend your thesis, because this leaves time to find your own funding for your ‘perfect’ post-doc if necessary.
-Never put all your eggs in one basket: even if a position seems a done deal, it can fall through.
-Your PhD supervisor can help to arrange a post-doc, but build your own connections too.
-A post-doc is a transition between a PhD and an academic position: you learn to write papers fast, respond to reviewers, and, ultimately, be a ‘true’ researcher
-There are varying degrees of flexibility: some supervisors assign you work, others let you work on your own. Make sure your offer is the right fit.
-Get a post doc at a different place than your PhD.  It could be your last chance to pick up and move somewhere for 1-2 years
-It’s hard to get a post-doc in a different subject than your PhD, but not impossible.  It’s a good time to move into another subject area if you want.

Impact Evaluation: We have cool methods! But need to work better with others.

We were privileged to have Jack Colford of UC Berkeley deliver a 4 hour (!) session on impact evaluation in the Neuroscience building yesterday.   All in all, Dr Colford’s insight was even more impressive than the list of letters next to his name.  He ran through applied examples of evaluating the impact of global health interventions in California, Mexico, Kenya, Bangladesh and India, to name a few (‘where hasn’t he done/run a study?’ was one of my unanswered questions).

If you missed it (or forgot a pen), some highlights of his insight included:

1. Stepped Wedge randomized controlled trials: In this design, communities are randomized to receive the intervention at each time interval, x. At x0, no community has it, at x1, some number of communities get ‘treated’, at x2, some more communities get it, and so on. In this way, all communities eventually get the intervention.  Further, all communities get to exist as both controls and treated groups at different time points (familiar to a case crossover?!), enabling some particularly interesting analyses (such as separating the effect in treatment compliers from communities who would have seen the effect regardless of treatment!)

2. Different options for randomization when you know baseline characteristics are dissimilar:
-Match similar pairs, then randomize 1 partner to receive the intervention, and 1 to be control
-‘Big stick’ randomization: let a computer create 10000’s of randomization sequences, and flag the sequences that meet your pre-specified ‘acceptable’ criterion for similar distributions of characteristics.  Of the resulting possible randomizations, you randomly choose 1.

3.  Retrospective impact evaluation: Sometimes (most of the time!), the program is already in place so it is not possible to implement it randomly or using stepped-wedge. In this case, communities can be assigned propensity scores, i.e. the probability of having received the intervention (which is based on a set of covariates you derive from various sources). These communities are compared against controls with similar propensity scores who happened to miss out on the intervention.  So you’re essentially ‘mimicking’ randomization in hindsight.

4. Collaboration is essential but flawed: Partners (NGOs, funders, governments and otherwise) are highly necessary, but it seems that differing reward systems hinder these collaborations’ potential:
-People at the World Bank are more highly rewarded for getting something done then actually doing something.
-People in NGOs are more highly rewarded for doing something then evaluating whether it was done right
-We, researchers, are more highly rewarded for publishing whether something was done right than disseminating this  vital information to stakeholders.
In other words, the system is a series of contra-indications (let me guess – as if the whole word wasn’t enough, it’s on us to fix this one too).

For more on Jack Colford’s work and methods:

Jack Colford
Paper illustrating stepped wedge RCT design
Methodological paper on estimating Complier Average Causal Effect in stepped wedge design
-Paper illustrating matching when intervention already exists
Methodological paper on matching when intervention already exists