DevOps AWS Big Data (Java)
World’s Most Comprehensive Cross-Device Identity big data on Businesses, Consumers and Geo Audiences. When the number of records counts hundreds of billions and datasets are terabytes on dozens of instances, every optimization counts — either performance, or cost, or disk space, or stability & backups — everything is critical.
- Perfect knowledge of Linux based OS and network protocols (iptables, ssh, HTTP, FTP, mail, LDAP, Samba, etc)
- Perfect knowledge of AWS products (Amazon ES, EC2, VPC, VPN/ssh tunnel, ELB, S3, EMR, IAM, and Security) and infrastructure
- Experience with Docker
- Any monitoring tools (DataDog, Nagios, CloudWatch, Zabbix, Ganglia, etc.)
- Ability to write bash scripts and automate infrastructure routine actions, writing from scratch and modifying config files
- Intermediate+ English (at least written)
- Team player
- A desire and ability of self-growth and self-education
- Knowledge of any programming language, preferably Java.
- Experience with Elasticsearch, Kibana, and Logstash
- Experience with big data projects
- Knowledge of any Infrastructure as a Code tools (AWS Cloud Formation, Terraform, etc.)
- Experience with log analysis tools (Splunk, Papertrail, etc.)
- Good understanding of managed and not managed NoSQL databases and key-value storages
- Knowledge of Jenkins or any other CI/CD tools
- Pro-active approach of work in a mindset
- Process oriented
- Attention to details and accuracy
- Knowledge of Azure and MS Server, IIS, MS SQL Server
- Experience of the Hadoop related stack (HDFS, YARN, HBase, Spark)
- Experience with Ansible
- Hands-on all AWS Re-Invent updates
- Ability to work overtime with the same performance
- Support several small and mid-size development teams with multiple AWS accounts and instances in different regions:
a) Set up and configure EC2 Linux machines and related packages (Docker, Java, PHP, node.js, Databases, etc.)
b) Set up a virtual private network and secure access in AWS
c) Set up monitoring and resolve incidents (out of space, out of memory, certificates expiration, etc.)
d) Set up and manage products AWS (Amazon ES, EC2, ELB, S3, EMR, etc.)
e) Write bash scripts and automate infrastructure routine actions
f) Use (where possible) Infrastructure as a Code tools (AWS Cloud Formation, Terraform, etc.)
g) Set up and maintain CI/CD tools (Jenkins, etc)
- AWS accounts cost optimization:
a) Always think how to solve the task in the most cost-efficient way.
b) Propose actions to reduce infrastructure cost for existing accounts and implement them once it’s approved
c) Follow all new AWS product announcements and suggest improvements to projects’ infrastructures
d) Study AWS products and related technologies (monitoring tools, log analysis tools, databases, etc) to be able to help the team with configuration and improvements
e) Monitor and keep tracking on abnormal cost spikes to answer the question why it happened and what could be done to avoid that in future
- Communicate with the client:
a) On projects with a dedicated DevOps position (full-time or part-time) timely communicate with the customer in emails in English.
b) In case of project infrastructure incident (going on or just resolved) participate in conf calls with customer biz or tech team.
c) Send planned maintenance email notifications.
d) Send emails on critical monitoring incidents’ fix status (sometimes with the investigated possible reason)