In today's fast-evolving digital environment, Data Engineers play a crucial role within software houses. Here, at STX Next, Data Engineering is one of the strategic departments that allow us to expand our offering. In this article, we delve into the duties and skills of our Data Engineers, as well as discover our customers' profiles, resulting in creating a definition of the profile of a Data Engineer working at STX Next.
Who’s the customer of our Data Engineering services?
Long story short, each Data Engineer here works on a single project. But to understand what working in the Data Engineering department is like, you first need to understand the needs of our clients and the types of projects we work on.
At STX Next, we provide services to companies across a range of industries, including FinTech, EdTech, FMCG, Healthcare and more. When it comes to their size, we work with a variety, ranging from large corporations to promising start-ups. And while the company size is obviously important, there is one more aspect that has a greater impact on a Data Engineer's day-to-day responsibilities—the company's data maturity. In that manner, we tend to work with companies that are in the first two stages: starting with data and scaling with data. That’s why, our projects can last from a few weeks or months (POC) to over a year, e.g. when we implement and maintain a more complex solution.
As for the amount of data, Big Data does not always mean a large amount of data in volume. Although there are projects where our Data Engineers process several TBs of data per month, more and more customers are processing relatively small amounts of data, instead focusing on its quality, proper cataloguing, ease of access and making it available to data consumers as quickly as possible. At the same time, they often need to source this data from different data sources.
Data Engineer’s tech stack and types of projects
The diversity of projects we participate in finds its reflection in the technological stack of our Data Engineers. Although we focus on working in the cloud and our team consists of people who specialise in AWS, Azure, GCP, Databricks or Snowflake, there are also projects that are implemented on-premises, e.g. when domain specific constraints do not allow the client to build cloud solutions. And while we always think in terms of choosing the "right tools for the right job" when designing solutions rather than creating CV-driven architectures, we try to keep up to date with trending technologies, evaluate them and apply them where they are useful. That’s why in our projects we’re using dbt, DuckDB and open data formats such as Apache Iceberg, Delta Lake or Hudi.
Typically, our projects fall into one of two categories:
- Designing and implementing solutions, such as data warehouse or data lake, from scratch - planning and overseeing data migration, creating pipelines to collect data (both batch and stream) from various sources such as databases, APIs, object stores, CRM systems, etc.
- Improving existing systems - analysing the current situation, identifying potential problems, customer needs and optimising processes by refactoring the architecture, applying best practices within the technologies already used, introducing data observability tools and a proper data pipeline monitoring system, etc.
Day to day work of STX Next’s Data Engineer
So, when we speak of an every-day work of a Data Engineer at STX Next, we’ve got to establish a few things first. As for the team—remember, you will never work alone on a project. We’re cooperating in groups of two, three or four, or if the project is small (e.g. a POC) the Data Engineer will be supported by a Data Architect.
As the projects vary in terms of team size, we do not impose a specific working methodology. However, it is safe to say that most projects use one of the frameworks that fall under the Agile methodology. So you may have daily meetings or just one meeting a week - it depends.
And just as projects, technologies and team sizes vary, so does the scope of responsibilities. Some typical examples are: implementing and maintaining ETL systems, integrating different data sources or solving problems related to their processing, conducting feasibility studies. What’s crucial is that our clients trust our engineers and rely on their experience and expertise, often giving them freedom of action and decision-making.
Once the project is finished, we’re awaiting to be assigned to another client. We can use this time for self-development by attending courses, obtaining certificates or writing articles.
It is worth mentioning that we support each other and share knowledge. Our team has Data Architects who advise and look after the development of other team members. We also do team bonding from time to time, as we like to meet as a whole team, no matter who is working on what project!
Can I be a Data Engineer at STX Next?
Summing it up, Data Engineers at STX Next are true masters of their craft. Their responsibilities differ based on the project they are a part of, but in the vast majority of cases they either start with the client’s data or help the client grow with data. Our Data Engineers master diverse tech: AWS, Azure, GCP, as well as various on-premises solutions. We prioritise pragmatism and continuous personal development. After finishing your project and waiting for a new one, you will have a chance to focus on yourself and your skills—we’re offering certificates, courses, and other activities to increase your competences.
So, how about joining our Data Engineering team? Check our job listing and see if we’re currently looking for someone who matches your profile!