Open (Data) Science – Data Management is Key

Psychology degrees do give you superpowers. No, you can’t read minds, but you can shut your eyes and perform t-tests on SPSS. There is a point in time where I can rely on muscle memory alone to get the p-values popping up my screen. I can see how for simple analysis point and click solutions can be much preferable.

It wasn’t until my final year project that I started to really recognise the importance of transparent and replicable codes and research processes. I had to co-collect data along with my peers, we each collect half of the data. The data collection process is by no means easy: a collage of cognitive tests, long structured interviews etc. keeping the participants (mainly other undergraduate students) engaged and put effort was always a challenge.

All is well until I discovered my data collection partner was popping in random numbers in the dataset. “What is she doing?!” Shocked I was seeing this, but even more shocked when it turned out that she forgot to record age and gender in the interviews. Not having gender is less relevant, but age is a key variable we have to take into account for the distribution of IQ differs by age. I was quite disappointed as this meant that the data was compromised, and I couldn’t bring myself to really trust what the data could tell me – if basic demographics are made up, how credible can other information be?

I think of this experience often as I read research papers that does not describe their data collection and analysis plan very well. In academia, more people are willing to share the codes they used for analysis. I believe the next step is to extend the transparency to data cleaning and management processes. When describing the process of data curation, it is easy to focus on describing the psychometric properties of the questions. However, I believe there are a lot of wisdom research groups can benefit from sharing their responses to more general questions like: How is the database managed? Were there any challenges to data management, how were they resolved? Studies and initaitives like PROSPER (PROfeSsionnal develoPmEnt for Research methodologists), helps consult and formulate future developments plans for research methodologists. I am happy to see how things are developing, and for methodologists to finally gain the limelight a bit more in research!

Till the end of the day, it is rare for any finding to be considered completely irrefutable or beyond scrutiny. The best we can do as researcher, in my opinion, is for future researchers to acknowledge as they read our findings, and say: “They were bound by the knowledge of their time, that was the most rigorous way they could have done it!”.

A key development goal from my PhD is the ability to develop codes and wrangling with data across Stata, R and Python. I am still learning the way to work across platforms that would make the most sense! Do share with me if you have any tips 🙂