That’s the second sizeable project boxed off, with work now complete on my Data Engineering with Azure and Databricks project.
I really enjoyed this one and learned a huge amount through the process. I’m now comfortable working in Databricks and applying the Bronze → Silver → Gold transformation pattern end to end. I’m also feeling increasingly confident with Python as each day passes. There is, of course, still a great deal to learn, but it’s definitely starting to feel more natural and embedded.
It was also good to revisit Tableau. It’s been around six years since I last used it, when I evaluated alternative BI platforms before ultimately settling on Power BI. I’ll admit I still prefer Power BI, but there’s undoubtedly some bias there given the many hours I’ve spent working with it. That said, it was useful to reacquaint myself with another BI tool and demonstrate cross-platform capability.
Daily upskilling in Python and SQL
Python has been a major focus over the past few weeks, particularly Pandas and PySpark. I’m currently enrolled on courses for both. I started with Pandas due to its strength in data manipulation, but it quickly became clear that for the data engineering project, PySpark was the more appropriate tool. That’s why I’ve chosen to work with both, understanding where each fits best.
Alongside this, SQL remains a daily practice area, particularly around data modelling and transformation logic.
Project #3 - v2 of the Job Insights Project
The next iteration of the Job Insights Project will focus on exploring the use of AI across the data pipeline and semantic model. Exactly how AI will be incorporated is still under consideration, but I’m keen to research and apply it thoughtfully rather than for novelty’s sake.
With the exception of MCP, which will be included, I’ve not yet had the opportunity to explore this area in depth. Given how quickly things are moving, staying current with AI capabilities is now essential for modern data roles, so this feels like the right next step.
CI/CD is also a prerequisite for many analyst, engineering, and BI roles. I’ve already incorporated CI/CD into my personal site to become familiar with the process, and I’m keen to explore how well these practices translate into Power BI and analytics workflows.
Looking ahead to Project #4 — Microsoft Fabric
It’s increasingly clear where Microsoft’s direction of travel is with data analytics: a move away from individual Azure-native services such as Data Factory and Synapse, towards a more unified experience through Microsoft Fabric.
From analysing the Job Insights data, most organisations are still operating primarily within Azure, but the shift toward Fabric is clearly underway, as reflected in the growing number of roles referencing the platform.
As a result, this will be the next major project in the portfolio. My current thinking is to support this work with the Microsoft DP-600 (Fabric Analytics Engineer Associate) certification.