Job Description
We are seeking a skilled Python Developer who will play a key role in building and optimizing Machine Learning pipelines for LLMs. You will be responsible for developing, deploying, and maintaining production-level ML pipelines that can handle large-scale AI applications. You will also work with Kubernetes to ensure that these models are deployed, scaled, and maintained efficiently in cloud environments.
Key Responsibilities:
- Design, develop, and optimize Machine Learning pipelines to automate the training, validation, and deployment of Large Language Models (LLMs).
- Collaborate with data scientists and ML engineers to ensure smooth integration of LLM models into production environments.
- Write scalable, efficient, and maintainable Python code for building robust ML pipelines and deploying LLMs.
- Work with Kubernetes to deploy, scale, and manage machine learning applications in cloud-based environments (e.g., AWS, GCP, or Azure).
- Implement model monitoring and management tools to ensure performance and reliability of deployed LLMs.
- Optimize the performance of ML pipelines, including preprocessing, model training, and serving.
- Ensure reproducibility, automation, and scalability across ML pipelines.
- Troubleshoot and resolve issues related to model deployment and performance in Kubernetes environments.
- Stay current with advancements in ML technologies, particularly around LLMs and Kubernetes, and apply this knowledge to enhance existing systems.
Required Skills and Qualifications:
- Strong experience with Python and Machine Learning frameworks (e.g.Hugging Face).
- Hands-on experience building and deploying Machine Learning pipelines in production environments.
- Expertise in deploying and managing applications using Kubernetes in cloud environments (AWS, GCP, Azure).
- Experience with containerization tools such as Docker.
- Familiarity with continuous integration/continuous deployment (CI/CD) processes for machine learning models.
- Solid understanding of Large Language Models (LLMs) and their application in NLP tasks (e.g., text generation, summarization, classification).
- Experience in automating the training, fine-tuning, and deployment of AI models.
- Strong knowledge of data preprocessing, feature engineering, and model evaluation techniques.
- Excellent problem-solving skills and the ability to work in a fast-paced environment.
- Strong communication skills and the ability to collaborate effectively within a cross-functional team.
- Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related field (advanced degrees are a plus).
Preferred Qualifications:
- Experience with cloud-native solutions for ML model deployment and orchestration (e.g., Kubernetes Operators, Helm).
- Familiarity with version control tools like Git.
- Knowledge of CI/CD pipelines for machine learning.
- Previous experience working in an Agile or startup environment.
- Experience with data storage solutions for machine learning (e.g., S3, BigQuery).