Data Science

TL;DR Causal inference is having a quiet renaissance in 2026 because the failure modes of correlation-only ML - distribution shift, recommendation feedback loops, biased hiring and lending models, broken A/B tests - have become too expensive to ignore The practical toolkit is econometrics come to industry: potential outcomes, DAGs, instrumental variables, regression discontinuity, synthetic controls, and double machine learning Tooling like DoWhy, EconML, and CausalML dropped the friction that used to keep these methods in academia, and AI assistants dropped the implementation cost The teams absorbing causal thinking are not abandoning predictive ML - they use predictive models where the world is stable and causal models where they are evaluating interventions The deeper shift: data science is moving from technical execution to analytical judgement, and judgement is the part that is not automatable For most of the deep-learning era, the answer to “why is this happening in our data?” was “we do not actually care - we care that our model predicts well.” For a wide range of problems, that pragmatism worked. The models did predict well. The business outcomes followed. The causal questions were left to academics and economists. ...

TL;DR Graph algorithms are memory-bound, not CPU-bound: a 5 billion node, 50 billion edge graph needs 500+ GB just for adjacency, scores, and working memory - it simply does not fit on one machine Four scaling strategies cover most production cases: distributed processing (vertex-cut or edge-cut partitioning), approximate algorithms (sampling, sketching, early stopping), incremental/streaming approaches, and storage-level optimisation (columnar layout, compression, caching) The hidden costs dominate at scale: communication overhead, synchronisation barriers, debugging distributed state, and the operational burden of keeping it all running Full-graph recompute is not feasible at billions of nodes - incremental algorithms and clever approximations are the norm, not the exception Managed services like Neptune Analytics remove much of the partitioning and operations work, at the price of less control Graph algorithms work great on your laptop. PageRank on a 100,000-node graph finishes in seconds. Louvain finds communities instantly. ...

TL;DR A community is a group of nodes with more internal connections than you would expect by chance; community detection finds these groups purely from network structure, not features Modularity is the metric everything uses - 0.3-0.7 is typical for real networks, and 0.9+ usually means you have over-clustered Louvain is the industry standard: fast, greedy, no hyperparameters to tune, and it produces high-modularity divisions; label propagation, spectral clustering, and Girvan-Newman fill the gaps The same technique powers friend-group detection, product clustering, fraud-ring identification, and finding functional modules in biological networks Overlapping communities remain the genuinely hard problem - most standard algorithms assign each node to exactly one group If you’ve ever looked at a social network and wondered “why is this group of people more connected to each other than to the rest of the network?”, you’ve just articulated the community detection problem. ...

TL;DR Graph queries answer relationship questions (paths, positions, influence) that are impossibly expensive in a relational database - the mental shift is from filtering rows to traversing paths PageRank measures a node’s importance by the importance of the nodes pointing to it; it is recursive, converges after 20-30 iterations, and applies far beyond web pages - citations, social influence, recommendations Centrality is an umbrella term: degree (how connected), betweenness (how many paths run through you), closeness (how near everything you are), and eigenvector (whether you are connected to important nodes) each answer a different question Picking the right centrality measure for the question matters more than tuning any single algorithm Managed engines like Neptune Analytics run these at scale without you implementing the iteration loops yourself Graph algorithms often get treated as academic curiosities - something you learn in a computer science course and then never think about again. But they’re actually the hidden backbone of some of the most profitable systems on the internet. ...