Data Engineer (Python Developer)
SQUARE ONE RESOURCES sp. z o.o.
Your responsibilities
- Create and maintain optimal data pipeline architecture.
- Assemble large, complex data sets that meet functional / non-functional business requirements.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS 'big data’ technologies.
- Development of CI/CD pipelines (esp. GitLab and DataOps)
- Modeling data transformations using dbt (data build tool), including scripting and complex algorithms
- Automation and simplification of processes (Bash, Docker, Python, GitLab, JIRA)
- Proficient use of AWS services, including but not limited to S3, Step Functions, Lambda, Fargate, AWS Glue, RDS. Strong hands-on experience with ETL (Extract, Transform, Load) processes, including the design and implementation of ETL solutions using AWS technologies.
- Strong experience in Python development, including writing clean, efficient, and maintainable code for data processing and automation tasks.
Our requirements
- Python (for data processing and automation tasks)
- AWS Services (S3, Step Functions, Lambda, Fargate, Glue, RDS)
- SQL (for data extraction, transformation, and loading)
- CI/CD & DataOps (experience with GitLab and building automation pipelines)
Aby ubiegać się o tę pracę, odwiedź stronę www.pracuj.pl.